{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Embedding Visualization\n",
    "\n",
    "In Tensorflow, data is represented by __tensors__ in our graph. Tensors are representetives for high dimensional data. For example MNIST images have $28\\times28=784$ dimensions, which are points in $\\mathbb{R}^{784}$ space. As humans, we are able to process our world which is a $\\mathbb{R}^{3}$ space ($x, y, z$). So, interpretation of the data is very hard in high dimensions.\n",
    "\n",
    "__Embeddin Visualization__ toolkit in Tensoarboard gives us the ability to reduce the dimension of our tensors using __Principle Component Analysis (PCA)__ or __t-Distributed Stochastic Neighbor Embedding (t-SNE)__ and visualize our data in lower dimension space.\n",
    "\n",
    "Let's build our Neural Net and try to visualize the tensors of activation of hidden layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports:\n",
    "We will start with importing the needed libraries for our code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# imports\n",
    "import os\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input data:\n",
    "For this tutorial we use the MNIST dataset. MNIST is a dataset of handwritten digits. If you are into machine learning, you might have heard of this dataset by now. MNIST is kind of benchmark of datasets for deep learning. One other reason that we use the MNIST is that it is easily accesible through Tensorflow. If you want to know more about the MNIST dataset you can check Yann Lecun's website.\n",
    "We can easily import the dataset and see the size of training, test and validation set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n",
      "Size of:\n",
      "- Training-set:\t\t55000\n",
      "- Test-set:\t\t10000\n",
      "- Validation-set:\t5000\n"
     ]
    }
   ],
   "source": [
    "# Import MNIST data\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)\n",
    "\n",
    "print(\"Size of:\")\n",
    "print(\"- Training-set:\\t\\t{}\".format(len(mnist.train.labels)))\n",
    "print(\"- Test-set:\\t\\t{}\".format(len(mnist.test.labels)))\n",
    "print(\"- Validation-set:\\t{}\".format(len(mnist.validation.labels)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyper-parameters:\n",
    "Hyper-parameters are important parameters which are not learned by the network. So, we have to specify them externally. These parameters are constant and they are not learnable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# hyper-parameters\n",
    "logs_path = \"./logs/embedding/\"  # path to the folder that we want to save the logs for Tensorboard\n",
    "learning_rate = 0.001  # The optimization learning rate\n",
    "epochs = 10  # Total number of training epochs\n",
    "batch_size = 100  # Training batch size\n",
    "display_freq = 100  # Frequency of displaying the training results\n",
    "\n",
    "# Network Parameters\n",
    "# We know that MNIST images are 28 pixels in each dimension.\n",
    "img_h = img_w = 28\n",
    "\n",
    "# Images are stored in one-dimensional arrays of this length.\n",
    "img_size_flat = img_h * img_w\n",
    "\n",
    "# Number of classes, one class for each of 10 digits.\n",
    "n_classes = 10\n",
    "\n",
    "# number of units in the first hidden layer\n",
    "h1 = 200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Graph:\n",
    "Like before, we start by constructing the graph. But, we need to define some functions that we need rapidly in our code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# weight and bais wrappers\n",
    "def weight_variable(name, shape):\n",
    "    \"\"\"\n",
    "    Create a weight variable with appropriate initialization\n",
    "    :param name: weight name\n",
    "    :param shape: weight shape\n",
    "    :return: initialized weight variable\n",
    "    \"\"\"\n",
    "    initer = tf.truncated_normal_initializer(stddev=0.01)\n",
    "    return tf.get_variable('W_' + name,\n",
    "                           dtype=tf.float32,\n",
    "                           shape=shape,\n",
    "                           initializer=initer)\n",
    "\n",
    "\n",
    "def bias_variable(name, shape):\n",
    "    \"\"\"\n",
    "    Create a bias variable with appropriate initialization\n",
    "    :param name: bias variable name\n",
    "    :param shape: bias variable shape\n",
    "    :return: initialized bias variable\n",
    "    \"\"\"\n",
    "    initial = tf.constant(0., shape=shape, dtype=tf.float32)\n",
    "    return tf.get_variable('b_' + name,\n",
    "                           dtype=tf.float32,\n",
    "                           initializer=initial)\n",
    "\n",
    "\n",
    "def fc_layer(x, num_units, name, use_relu=True):\n",
    "    \"\"\"\n",
    "    Create a fully-connected layer\n",
    "    :param x: input from previous layer\n",
    "    :param num_units: number of hidden units in the fully-connected layer\n",
    "    :param name: layer name\n",
    "    :param use_relu: boolean to add ReLU non-linearity (or not)\n",
    "    :return: The output array\n",
    "    \"\"\"\n",
    "    with tf.variable_scope(name):\n",
    "        in_dim = x.get_shape()[1]\n",
    "        W = weight_variable(name, shape=[in_dim, num_units])\n",
    "        tf.summary.histogram('W', W)\n",
    "        b = bias_variable(name, [num_units])\n",
    "        tf.summary.histogram('b', b)\n",
    "        layer = tf.matmul(x, W)\n",
    "        layer += b\n",
    "        if use_relu:\n",
    "            layer = tf.nn.relu(layer)\n",
    "        return layer\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our helper functions we can create our graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create graph\n",
    "# Placeholders for inputs (x), outputs(y)\n",
    "with tf.variable_scope('Input'):\n",
    "    x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='X')\n",
    "    tf.summary.image('input_image', tf.reshape(x, (-1, img_w, img_h, 1)), max_outputs=5)\n",
    "    y = tf.placeholder(tf.float32, shape=[None, n_classes], name='Y')\n",
    "fc1 = fc_layer(x, h1, 'Hidden_layer', use_relu=True)\n",
    "output_logits = fc_layer(fc1, n_classes, 'Output_layer', use_relu=False)\n",
    "\n",
    "# Define the loss function, optimizer, and accuracy\n",
    "with tf.variable_scope('Train'):\n",
    "    with tf.variable_scope('Loss'):\n",
    "        loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=output_logits), name='loss')\n",
    "        tf.summary.scalar('loss', loss)\n",
    "    with tf.variable_scope('Optimizer'):\n",
    "        optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, name='Adam-op').minimize(loss)\n",
    "    with tf.variable_scope('Accuracy'):\n",
    "        correct_prediction = tf.equal(tf.argmax(output_logits, 1), tf.argmax(y, 1), name='correct_pred')\n",
    "        accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32), name='accuracy')\n",
    "        tf.summary.scalar('accuracy', accuracy)\n",
    "        # Network predictions\n",
    "        cls_prediction = tf.argmax(output_logits, axis=1, name='predictions')\n",
    "\n",
    "# Initializing the variables\n",
    "init = tf.global_variables_initializer()\n",
    "merged = tf.summary.merge_all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training epoch: 1\n",
      "iter   0:\t Loss=2.28,\tTraining Accuracy=37.0%\n",
      "iter 100:\t Loss=0.45,\tTraining Accuracy=85.0%\n",
      "iter 200:\t Loss=0.21,\tTraining Accuracy=95.0%\n",
      "iter 300:\t Loss=0.34,\tTraining Accuracy=89.0%\n",
      "iter 400:\t Loss=0.29,\tTraining Accuracy=90.0%\n",
      "iter 500:\t Loss=0.18,\tTraining Accuracy=97.0%\n",
      "---------------------------------------------------------\n",
      "Epoch: 1, validation loss: 0.21, validation accuracy: 94.1%\n",
      "---------------------------------------------------------\n",
      "Training epoch: 2\n",
      "iter   0:\t Loss=0.14,\tTraining Accuracy=97.0%\n",
      "iter 100:\t Loss=0.19,\tTraining Accuracy=94.0%\n",
      "iter 200:\t Loss=0.11,\tTraining Accuracy=95.0%\n",
      "iter 300:\t Loss=0.06,\tTraining Accuracy=99.0%\n",
      "iter 400:\t Loss=0.08,\tTraining Accuracy=96.0%\n",
      "iter 500:\t Loss=0.12,\tTraining Accuracy=97.0%\n",
      "---------------------------------------------------------\n",
      "Epoch: 2, validation loss: 0.14, validation accuracy: 96.0%\n",
      "---------------------------------------------------------\n",
      "Training epoch: 3\n",
      "iter   0:\t Loss=0.16,\tTraining Accuracy=96.0%\n",
      "iter 100:\t Loss=0.15,\tTraining Accuracy=96.0%\n",
      "iter 200:\t Loss=0.08,\tTraining Accuracy=97.0%\n",
      "iter 300:\t Loss=0.11,\tTraining Accuracy=94.0%\n",
      "iter 400:\t Loss=0.10,\tTraining Accuracy=96.0%\n",
      "iter 500:\t Loss=0.09,\tTraining Accuracy=97.0%\n",
      "---------------------------------------------------------\n",
      "Epoch: 3, validation loss: 0.11, validation accuracy: 96.7%\n",
      "---------------------------------------------------------\n",
      "Training epoch: 4\n",
      "iter   0:\t Loss=0.03,\tTraining Accuracy=100.0%\n",
      "iter 100:\t Loss=0.04,\tTraining Accuracy=99.0%\n",
      "iter 200:\t Loss=0.11,\tTraining Accuracy=94.0%\n",
      "iter 300:\t Loss=0.05,\tTraining Accuracy=98.0%\n",
      "iter 400:\t Loss=0.11,\tTraining Accuracy=97.0%\n",
      "iter 500:\t Loss=0.11,\tTraining Accuracy=95.0%\n",
      "---------------------------------------------------------\n",
      "Epoch: 4, validation loss: 0.09, validation accuracy: 97.2%\n",
      "---------------------------------------------------------\n",
      "Training epoch: 5\n",
      "iter   0:\t Loss=0.11,\tTraining Accuracy=97.0%\n",
      "iter 100:\t Loss=0.04,\tTraining Accuracy=98.0%\n",
      "iter 200:\t Loss=0.04,\tTraining Accuracy=100.0%\n",
      "iter 300:\t Loss=0.02,\tTraining Accuracy=100.0%\n",
      "iter 400:\t Loss=0.11,\tTraining Accuracy=97.0%\n",
      "iter 500:\t Loss=0.04,\tTraining Accuracy=99.0%\n",
      "---------------------------------------------------------\n",
      "Epoch: 5, validation loss: 0.08, validation accuracy: 97.7%\n",
      "---------------------------------------------------------\n",
      "Training epoch: 6\n",
      "iter   0:\t Loss=0.04,\tTraining Accuracy=100.0%\n",
      "iter 100:\t Loss=0.07,\tTraining Accuracy=98.0%\n",
      "iter 200:\t Loss=0.08,\tTraining Accuracy=98.0%\n",
      "iter 300:\t Loss=0.04,\tTraining Accuracy=99.0%\n",
      "iter 400:\t Loss=0.04,\tTraining Accuracy=99.0%\n",
      "iter 500:\t Loss=0.05,\tTraining Accuracy=97.0%\n",
      "---------------------------------------------------------\n",
      "Epoch: 6, validation loss: 0.07, validation accuracy: 97.9%\n",
      "---------------------------------------------------------\n",
      "Training epoch: 7\n",
      "iter   0:\t Loss=0.04,\tTraining Accuracy=99.0%\n",
      "iter 100:\t Loss=0.03,\tTraining Accuracy=99.0%\n",
      "iter 200:\t Loss=0.07,\tTraining Accuracy=99.0%\n",
      "iter 300:\t Loss=0.06,\tTraining Accuracy=98.0%\n",
      "iter 400:\t Loss=0.13,\tTraining Accuracy=96.0%\n",
      "iter 500:\t Loss=0.07,\tTraining Accuracy=97.0%\n",
      "---------------------------------------------------------\n",
      "Epoch: 7, validation loss: 0.07, validation accuracy: 97.8%\n",
      "---------------------------------------------------------\n",
      "Training epoch: 8\n",
      "iter   0:\t Loss=0.03,\tTraining Accuracy=99.0%\n",
      "iter 100:\t Loss=0.01,\tTraining Accuracy=100.0%\n",
      "iter 200:\t Loss=0.03,\tTraining Accuracy=99.0%\n",
      "iter 300:\t Loss=0.02,\tTraining Accuracy=100.0%\n",
      "iter 400:\t Loss=0.02,\tTraining Accuracy=100.0%\n",
      "iter 500:\t Loss=0.05,\tTraining Accuracy=98.0%\n",
      "---------------------------------------------------------\n",
      "Epoch: 8, validation loss: 0.07, validation accuracy: 97.8%\n",
      "---------------------------------------------------------\n",
      "Training epoch: 9\n",
      "iter   0:\t Loss=0.04,\tTraining Accuracy=99.0%\n",
      "iter 100:\t Loss=0.01,\tTraining Accuracy=100.0%\n",
      "iter 200:\t Loss=0.07,\tTraining Accuracy=98.0%\n",
      "iter 300:\t Loss=0.03,\tTraining Accuracy=99.0%\n",
      "iter 400:\t Loss=0.03,\tTraining Accuracy=99.0%\n",
      "iter 500:\t Loss=0.09,\tTraining Accuracy=97.0%\n",
      "---------------------------------------------------------\n",
      "Epoch: 9, validation loss: 0.07, validation accuracy: 97.9%\n",
      "---------------------------------------------------------\n",
      "Training epoch: 10\n",
      "iter   0:\t Loss=0.02,\tTraining Accuracy=100.0%\n",
      "iter 100:\t Loss=0.03,\tTraining Accuracy=100.0%\n",
      "iter 200:\t Loss=0.02,\tTraining Accuracy=100.0%\n",
      "iter 300:\t Loss=0.02,\tTraining Accuracy=100.0%\n",
      "iter 400:\t Loss=0.01,\tTraining Accuracy=100.0%\n",
      "iter 500:\t Loss=0.01,\tTraining Accuracy=100.0%\n",
      "---------------------------------------------------------\n",
      "Epoch: 10, validation loss: 0.07, validation accuracy: 98.1%\n",
      "---------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Launch the graph (session)\n",
    "sess = tf.InteractiveSession() # using InteractiveSession instead of Session to test network in separate cell\n",
    "sess.run(init)\n",
    "train_writer = tf.summary.FileWriter(logs_path, sess.graph)\n",
    "num_tr_iter = int(mnist.train.num_examples / batch_size)\n",
    "global_step = 0\n",
    "for epoch in range(epochs):\n",
    "    print('Training epoch: {}'.format(epoch + 1))\n",
    "    for iteration in range(num_tr_iter):\n",
    "        batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
    "        global_step += 1\n",
    "        # Run optimization op (backprop)\n",
    "        feed_dict_batch = {x: batch_x, y: batch_y}\n",
    "        _, summary_tr = sess.run([optimizer, merged], feed_dict=feed_dict_batch)\n",
    "        train_writer.add_summary(summary_tr, global_step)\n",
    "\n",
    "        if iteration % display_freq == 0:\n",
    "            # Calculate and display the batch loss and accuracy\n",
    "            loss_batch, acc_batch = sess.run([loss, accuracy],\n",
    "                                             feed_dict=feed_dict_batch)\n",
    "            print(\"iter {0:3d}:\\t Loss={1:.2f},\\tTraining Accuracy={2:.01%}\".\n",
    "                  format(iteration, loss_batch, acc_batch))\n",
    "\n",
    "    # Run validation after every epoch\n",
    "    feed_dict_valid = {x: mnist.validation.images, y: mnist.validation.labels}\n",
    "    loss_valid, acc_valid = sess.run([loss, accuracy], feed_dict=feed_dict_valid)\n",
    "    print('---------------------------------------------------------')\n",
    "    print(\"Epoch: {0}, validation loss: {1:.2f}, validation accuracy: {2:.01%}\".\n",
    "          format(epoch + 1, loss_valid, acc_valid))\n",
    "    print('---------------------------------------------------------')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create embeddings\n",
    "\n",
    "Let's say we are interested in visualizing the tensor of activation of the hidden layer nodes in the test test.\n",
    "\n",
    "In this example our test set has 10000 samples. Our hidden layer has $200$ nodes. So, the output tensor of hidden layer has a shape of 10000$\\times$200.\n",
    "\n",
    "We will create an embedding variable with the shape (10000 , 200) and assing the of activation of the hidden layer ```(fc1)``` to the variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Load the test set\n",
    "x_test = mnist.test.images\n",
    "y_test = mnist.test.labels\n",
    "\n",
    "# Initialize the embedding variable with the shape of our desired tensor\n",
    "tensor_shape = (x_test.shape[0] , fc1.get_shape()[1].value) # [test_set , h1] = [10000 , 200]\n",
    "embedding_var = tf.Variable(tf.zeros(tensor_shape), \n",
    "                            name='fc1_embedding')\n",
    "# assign the tensor that we want to visualize to the embedding variable\n",
    "embedding_assign = embedding_var.assign(fc1) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it is time to call the embedding functions to write the information in the memory.\n",
    "\n",
    "\n",
    "Remember, our goal is to visualize a tensor of high dimensional data, in a low dimension to be understandable for human. In order to understand the corresponding image and label ofeach point in the low dimension, we have two files:\n",
    "\n",
    "1. __metadata.tsv__: to store index and label of each sample\n",
    "2. __sprite_image.png__: to store all the sample images in a very large image \n",
    "\n",
    "The tensor that we are trying to visualize is a 10000$\\times$200. We have 10000 samples. So, we need a __metadata.tsv__ file that stores 10000 indices of the images and their corresponding labels. We will also need to create a large image with containing 10000 small MNIST images (with size 28$\\times$28).\n",
    "\n",
    "Without introducing the __metadata__ and __sprite image__ our visualization is nothing but some points with indices:\n",
    "\n",
    "<img src=\"https://github.com/easy-tensorflow/easy-tensorflow/raw/master/4_Tensorboard/Tutorials/files/embedding_no_metadata.gif\">\n",
    "\n",
    "So we will add the __metadata__ and __sprite image__ information to our config file.\n",
    "\n",
    "We will create this files later.  But in the embedding config file, we have to specify the path to these files.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tensorflow.contrib.tensorboard.plugins import projector\n",
    "\n",
    "# Create a config object to write the configuration parameters\n",
    "config = projector.ProjectorConfig()\n",
    "\n",
    "# Add embedding variable\n",
    "embedding = config.embeddings.add()\n",
    "embedding.tensor_name = embedding_var.name\n",
    "\n",
    "# Link this tensor to its metadata file (e.g. labels) -> we will create this file later\n",
    "embedding.metadata_path = 'metadata.tsv'\n",
    "\n",
    "# Specify where you find the sprite. -> we will create this image later\n",
    "embedding.sprite.image_path = 'sprite_images.png'\n",
    "embedding.sprite.single_image_dim.extend([img_w, img_h])\n",
    "\n",
    "# Write a projector_config.pbtxt in the logs_path.\n",
    "# TensorBoard will read this file during startup.\n",
    "projector.visualize_embeddings(train_writer, config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run our session to evaluate the tensor for embedding and save the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./logs/embedding/model.ckpt-5500'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Run session to evaluate the tensor\n",
    "x_test_fc1 = sess.run(embedding_assign, feed_dict={x: x_test})\n",
    "\n",
    "# Save the tensor in model.ckpt file\n",
    "saver = tf.train.Saver()\n",
    "saver.save(sess, os.path.join(logs_path, \"model.ckpt\"), global_step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## create metadata and sprite images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def write_sprite_image(filename, images):\n",
    "    \"\"\"\n",
    "        Create a sprite image consisting of sample images\n",
    "        :param filename: name of the file to save on disk\n",
    "        :param shape: tensor of flattened images\n",
    "    \"\"\"\n",
    "\n",
    "    # Invert grayscale image\n",
    "    images = 1 - images\n",
    "\n",
    "    # Calculate number of plot\n",
    "    n_plots = int(np.ceil(np.sqrt(images.shape[0])))\n",
    "\n",
    "    # Make the background of sprite image\n",
    "    sprite_image = np.ones((img_h * n_plots, img_w * n_plots))\n",
    "\n",
    "    for i in range(n_plots):\n",
    "        for j in range(n_plots):\n",
    "            img_idx = i * n_plots + j\n",
    "            if img_idx < images.shape[0]:\n",
    "                img = images[img_idx]\n",
    "                sprite_image[i * img_h:(i + 1) * img_h,\n",
    "                j * img_w:(j + 1) * img_w] = img\n",
    "\n",
    "    plt.imsave(filename, sprite_image, cmap='gray')\n",
    "    print('Sprite image saved in {}'.format(filename))\n",
    "\n",
    "\n",
    "def write_metadata(filename, labels):\n",
    "    \"\"\"\n",
    "            Create a metadata file image consisting of sample indices and labels\n",
    "            :param filename: name of the file to save on disk\n",
    "            :param shape: tensor of labels\n",
    "    \"\"\"\n",
    "    with open(filename, 'w') as f:\n",
    "        f.write(\"Index\\tLabel\\n\")\n",
    "        for index, label in enumerate(labels):\n",
    "            f.write(\"{}\\t{}\\n\".format(index, label))\n",
    "\n",
    "    print('Metadata file saved in {}'.format(filename))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sprite image saved in ./logs/embedding/sprite_images.png\n",
      "Metadata file saved in ./logs/embedding/metadata.tsv\n"
     ]
    }
   ],
   "source": [
    "# Reshape images from vector to matrix\n",
    "x_test_images = np.reshape(np.array(x_test), (-1, img_w, img_h))\n",
    "# Reshape labels from one-hot-encode to index\n",
    "x_test_labels = np.argmax(y_test, axis=1)\n",
    "\n",
    "write_sprite_image(os.path.join(logs_path, 'sprite_images.png'), x_test_images)\n",
    "write_metadata(os.path.join(logs_path, 'metadata.tsv'), x_test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we are finished the testing, we will close the session to free the memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# close the session after you are done with testing\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this step our coding is done. We have also plotted the accuarcy and some examples. But to inspect more in our network, we can run the __Tensorboard__. Open your terminal and type:\n",
    "```bash\n",
    "tensorboard --logdir=logs/embedding/ --host localhost\n",
    "```\n",
    "and Open the generated link in your browser.\n",
    "\n",
    "You can see the visualized tensor in __Projector__ tab:\n",
    "\n",
    "<img src=\"https://github.com/easy-tensorflow/easy-tensorflow/raw/master/4_Tensorboard/Tutorials/files/embedding.gif\">\n",
    "\n",
    "\n",
    "__NOTE:__ Don't forget to activate your environment !!!\n",
    "\n",
    "__NOTE:__ Sometimes you might not see the __Projector__ tab on top of the page. Try reloading your page and it will show up."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks for reading! If you have any question or doubt, feel free to leave a comment in our [website](http://easy-tensorflow.com/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
